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A Computer User Interface Using a Physical Manipulatory Grammar 

- ~~ # . 

Field of the Invention 

5 

The present invention relates to a method for supporting a hand or touch 
operated user interface. More particularly, the present invention relates to a method for 
interpreting user input based on physical manipulations. 

10 

Background and Summary of the Invention 

€1 Reliably, quickly, and intuitively transmitting complex commands to small 

i2 portable computers can be difficult. Small computer devices do not generally have 

y 15 sufficient computer processing power to respond reliably to voice or handwritten (pen 

m based) commands, keyboards are often absent or too small for accurate finger input, 

m and conventional buttons are too large or support too limited a command instruction 

H set. User interface techniques that rely on bulky external modules (full size infrared 

K linked keyboards, tethered data gloves, or camera based gestural recognition 

PU 20 equipment, for example) are expensive, often not readily available outside selected 

O sites, and probably too awkward for widespread use in conjunction with consumer level 
portable computing devices. 

User interface designers for portable computers have attempted to compensate 
25 for some of these problems by constructing devices that rely on various spatial, 
positional, or environmental cues that manually or automatically allow for activation of 
various modes in the device. For example, some laptop computers use the action of 
opening/closing the lid to initiate automatic bootup/powerdown of the computer without 
requiring any additional signal input (e.g. such as depressing a "start" button or typing 
30 "Lo_g_o_n" on a keyboard) from a user. Alternatively, the use of small portable 
computers that automatically switch control modes depending on position, orientation, 
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or room location have been investigated. Buttoniess manual control of a portable 
computer through deliberate user actions such as tilting the portable computer have 
also been described. 

5 However, all these solutions for interfacing with small portable computers have 

generally been limited in scope and functionality. What is needed is a user interface 
system suitable for even very small portable computers (having volumetric dimensions 
on the order of a one cubic centimeter) that is powerful, can be intuitively operated by 
an ordinary user with little training, and is still readily capable of modification or 

10 extension by the user. The present invention meets these requirements by providing a 
manipulatory user interface that responds to a user twisting, folding, bending, 
squeezing, shaking, tilting, spinning, lifting, or otherwise physically manipulating the 
computer. 

15 In the manipulatory user interface system of the present invention, the most 

basic level of manipulation is known as a 'sensemeU A senseme is defined as a single? 
indivisible type of physical manipulation. A partial list of categories of sensemes include 
material transformations such as squeezing, twisting, stretching; local spatial 
transformations such as translation, rotation, orbiting; and environmental 

20 transformations based on temperature, light level, or vibration. For example, a small 
portable computer may support a deformable piece having multiple embedded sensors 
that detect folding, twisting, or bending of the deformable piece by a user. This 
computer can also contain a number of accelerometers that sense relative spatial 
information; gyroscopic, radio or infrared positional sensors for determining absolute 

25 position; and various thermal or photosensors that respectively detect temperature and 
light level changes. Intentional or unintentional modifications detected by one or more 
of these sensor systems can provide the basis for a powerful user interface scheme. 

As will be appreciated, each senseme category contains many individually 
30 distinguishable members. For example, consider the category of senseme known as a 
"pinch", a structural transformation generally completed by a user squeezing the 
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definable piece between a forefinger and thumb. A pinch can be modified by varying 
its speed (quick or slow pinch), magnitude/intensity (light or hard pinch), portion of 
deferable piece pinched (top, bottom, or center of deformable piece pinched), or even 
portin of body used to pinch (right handed pinch or left handed pinch), with each 
moolcation being distinguishable as a senseme capable of being mapped onto a 
computer control command. 

Although the wide variety of easily distinguishable sensemes would alone 
provide a powerful user interface to a computer, the present invention further extends 
the iexibilrty of the senseme based user interface by supporting computer control 
based on a "morpheme" input. The morpheme is a temporally synchronous (or* 
overtyping asynchronous) tuple of one or more sensemes. Note that a morpheme can 
(and often will) contain more than one senseme. The sensemes combined into a 
mon*eme can come either from the same category (the user pinches with a right hand 
white tapping with a left hand finger), or different categories (the user pinches the 
deferable piece with a right hand while modifying the spatial position of the portable 
computer by tilting it forward). 

Any morpheme can in turn be extended by participation in a "sentence". A 
sentence is defined as a sequence of one or more temporally disjoint morphemes. The 
sentence level allows definition of a physical manipulatory grammar by appropriate 
choice of morpheme sequence, and corollary rules governing, for example, use of 
active (verb like) morphemes, naming (noun) morphemes, or connectors. Other 
possfcle grammar constructs used in sentences may include those based on "home" 
systems. Home systems are general-purpose gestural languages, whose grammar and 
syntax are not borrowed in any way from a host language. Examples of these 
languages are gestural languages developed by deaf children of hearing parents who 
have not been exposed to American Sign Language (ASL), and the "plains talk" of 
North American Indians, which was used as a trade language. 
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Accordingly, the present invention provides a method for inputting information to 
a computer connected to a deformable piece that can be manipulated, and optionally to 
various position sensors (both relative and absolute), pressure sensors, thermal 
sensors, or even light sensors. The method comprises the steps of manipulating the 

5 deformable piece to provide a first morpheme input to the computer, with the first 
morpheme input normally triggering a first default action by the computer. The 
deformable piece may also be manipulated to provide a second morpheme input to the 
computer, with the second morpheme input converting the normally triggered first 
default action to a second action. The first and second morphemes (and any 

io subsequent morphemes) together form a sentence that can be interpreted as a 
command to implement a computer controlled action, whether it be to unlock an 
electronically controlled door, display a graphical image on a computer display, or begin 
logging on to a computer network. Advantageously, such a user interface system is well 
suited for interaction with small computer devices, and may even be cross-cultural to a 

15 limited extent, with ability to squeeze or pinch being universal human attributes. 

The present invention is particularly useful for portable computers that can be 
held in the palm of a hand. If the portable computer is partially or completely 
surrounded with a deformable material having embedded or contact 

20 pressure/deformation sensors, a user is able to gesture with or manipulate the whole 
computer in order to achieve some desired result. Material deformation can be 
implemented at various scales. For instance, a computer and connected deformation 
sensors mounted inside a flexible tube could respond to right angle bending of the tube, 
or even complex knotting or looping of the tube. In practice however, only minute 

25 surface deformations are required, being just sufficient to provide tactile feedback to 
pressing, pinching, or bending type manipulations. In either case, however, the 
measurement of location and pressure applied to a surface is sufficient to characterize 
the mode of interaction (distinguishing a pinch from a prod). 

30 One particularly preferred embodiment of a handheld portable computer that 

responds to a physical manipulatory grammar in accordance with the present invention 
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includes a computer, a feedback module to provide visual, auditory, or tactile feedback 
to a user (e.g., processor connected LCD display, audio speaker, or tactile display to 
present Braille or other conventional touch interface), and co-mounted graspable, 
deformable piece partially or completely surrounding the feedback module. In addition, 
various thermal or pressure sensors are mounted to detect handedness of a user, with 
the grasping hand generally being the non-dominant hand for the user. Depending on 
the handedness of the user (which can be considered as a morpheme), the displayed 
data structure is modified. For example, text displayed on an LCD display screen may 
be automatically shifted rightward on the screen to allow pen based annotation on the 
left side of the screen, thereby aiding left handed users. 

Physically manipulatable user interfaces additionally provide an opportunity for 
multiple devices to interact in a user friendly manner. For example, a tileable display 
system having multiple displays with touch sensitive deformable pieces surrounding the 
display can be used to organize documents based on relative position of contact of 
each display. For example, if two displays initially showing separate data structures (e.g 
two different pages from two different electronic books) are brought together in side by 
side touching contact, the displayed visual information can alter (e.g. by displaying 
adjacent pages of a single electronic book). As will be appreciated, one can also use 
multiple computers connected to physically manipulatable control elements to construct 
complex commands for organizing data structures. 

Additional functions, objects, advantages, and features of the present invention 
will become apparent from consideration of the following description and drawings of 
preferred embodiments. 
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Brief Description of the Drawings 

Figure 1 is a schematic diagram of an approximately spherical hand holdable 
portable computer having a deformable surface, a status display, and a pressure 
s sensor array for detecting surface deformations; 

Figure 2 is a graphical diagram illustrating various physical manipulation 
morphemes, with axes respectively illustrating complexity of senseme tuples required to 
form a morpheme, and plasticity of a device required to support the physical 
10 manipulation; 

Figures 3-16 schematically illustrate various preferred classes of physical 
manipulation morphemes; 

15 Figure 17 is a graphical diagram illustrating various spatial morphemes, with a 

first axis illustrating complexity of senseme tuples required to form a morpheme, and a 
second axis illustrating the degree of positional information needed to support the 
physical manipulation (moving along a continuum from relative local measurement 
along a single specified dimension to absolute global measurement with six degrees of 

20 freedom determined); 

Figures 18-26 schematically illustrate various preferred spatial manipulation 
morphemes; 

25 Figure 27 is a graphical illustration showing increasing degrees of sensor system 

complexity that can be used to detect various categories of environmental stimuli, 
including light effects, thermal effects, the electromagnetic environment, and the 
vibratory/acoustic environment; 

30 Figure 28 is a graphical diagram illustrating various physical manipulation 

morphemes for multiple interacting devices, with axes respectively illustrating the 
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complexity of senseme tuples required to form a morpheme, and increasing level of 
physical contact; 

Figures 29-34 schematically illustrate various preferred manipulation morphemes 
5 for multiple interacting devices; 

Figures 35 and 36 are schematic diagrams illustrating "squeeze" and "tilt" 
morphemes applicable to a portable computer; 

10 Figure 37 is a schematic diagram illustrating tilt and squeeze morphemes used 

to control view of large two dimensional data sets with a relatively small display of a 
portable computer; 

Figure 38 is a schematic illustration representing a portable computer having a 
display ready to receive annotations from a right handed user; 

Figure 39 is a schematic illustration representing a portable computer having a 
display ready to receive annotations from a left handed user; 

Figure 40 is an electronic schematic illustrating components of the pressure and 
tilt sensitive modules of a portable computer such as illustrated in Figures 35-39. 

Figures 41 and 42 are schematic illustrations of a scanner/printer/copier using a 
paper shaped display interface to support morphemic input; 

Figures 43-45 are schematic illustrations of tilable displays capable of supporting 
morphemic input; 

Figure 46 illustrates optical sensors and patterns suitable for use in conjunction 
30 with tileable displays such as illustrated in Figures 43-45; 
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Figure 47 illustrates radio transponders suitable for use in conjunction with 
tileable displays such as illustrated in Figures 43-45; and 

Figure 48 illustrates addressing of multiple tileable displays. 
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Detailed Description of the Drawings 

Figure 1 illustrates an embodiment of the present invention suitable for 
supporting a morphemic user interface grammar. Support of the grammar can require 
detection of a user's physical manipulation of a device, detection of relative or absolute 
spatial location of the device, detection of various environmental factors acting on the 
device, and even detection and interaction with multiple devices or external computer 
networks. As illustrated, a device 10 has a deformable surface 20 with an underlying 
deformation sensor mesh 22 for detecting surface deformation across or within multiple 
subregions of the deformable surface 20. The deformation sensor mesh 22 is 
connected to an internally contained processor 24 having associated memory 
system 26. For detecting various positional or environmental variables, a sensing 
system 28 is also provided. The illustrated device further includes a feedback 
module 33, which may include an externally visible status display 30 or a non-visual 
feedback module 31 (typically delivering auditory or tactile feedback). In the illustrated 
device, a communications system 32 for reception or transmission of information to 
other electronic or computing devices is also provided. All these components can be 
powered by a power supply 25, which is usually an internally mounted rechargeable 
battery of conventional construction. 

Although the device 10 is illustrated as having an approximately spheroidal and 
unitary mass, various other shapes are contemplated to be within the scope of the 
present invention. For example, the overall shape may be similar to various rectangular 
prisms, or can be ellipsoidal, toroidal, planar, or even be malleable enough to support a 
wide range of user defined irregular shapes. In addition, multiple cooperating shape 
elements are contemplated using conventional designs that permit interlocking of 
multiple shape elements (e.g using a ball and socket, a lock and key, or siidable or 
rotatable interlocked components). 

Whatever the shape of device 10, for operation of the present invention the 
device 10 is completely or partially enveloped by the deformable surface 20. The 
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present invention supports use of a great variety of designs and materials for the 
deformable surface 20, depending on the required plasticity, durability,, longevity, and of 
course, cost constraints. For example, contemplated designs for deformable surface 20 
include, but are not limited to: 
5 a closed or open celled polymeric foam material having a wall thickness of 

millimeters to centimeters, with thinner walled embodiments being supported (e.g. by 
adhesive attachment) by an internal hard shell (constructed from polymeric or metallic 
materials), and those thicker walled embodiments directly supporting (by, e.g. brackets 
or supports) internal components such as processor 24. Suitable foams may include 

io those composed in whole or in part of widely available synthetic rubbers such as 
polychloroprene (neoprene), polystyrenes, rubber or nitrite rubber latex foams, 
polysiloxanes, block polymers including styrene-butadiene or styrene isoprene, or any 
other conventional material having good elasticity and deformability; 

a thin single layer polymeric surface loosely wrapped around a internal hard shell 

15 (the hard shell being constructed from polymeric or metallic materials). For example, a 
nylon or cotton weave, single layer polyethylene, synthetic rubber (with little or no foam 
cells present), or natural polymeric materials such as leather wrapped around a 
polystyrene casing can be used; 

a composite layered surface having a durable polymeric outer layer supported by 

20 an inner foam layer; or even 

a polymeric bilayer having an intermediate fluid or gel layer of a viscous or 
thixotropic material that can be used to support extreme deformations. The intermediate 
layer can be relatively thick (one the order of centimeters), or in certain embodiments 
can have a thickness measured on micron to millimeter scales. Such extremely thin 

25 layers would allow complex twisting, folding, curling, or crumpling actions, and have 
been described in conjunction with U.S. Patent 5,389,945, assigned to Xerox Corp., the 
disclosure of which is herein specifically incorporated by reference. 

The deformation sensor mesh 22 can be embedded within, or positioned to 
30 contact, the deformable surface 20. The deformation sensor mesh 22 can include an 
array of individual compressional or tensional strain sensors, or alternatively, embedded 

10 
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or attached positional sensors. For certain applications, continuous sensors (e.g. bilayer 
sheets of capacitance sensors) may be employed. One particularly useful, continuous 
sensor type uses multiple capacitance or resistance strips, with deformation pressure 
resulting in a positionally localizable analog signal proportional to the applied 
5 deformation pressure. Various sensor types can be used, including simple capacitance 
sensors, resistive strain sensors, analog or digital pressure switches, inductive sensors, 
or even fluid flow sensors. Depending on the sensor type employed, sensor data can 
be directly fed to the processor 24 in digital form, or be transformed to digital format by 
an general purpose analog/digital converter that typically provides a 4 or 8 bit range 
10 (although as few as one or as many as 32 bits may be required by various 
applications). An analog to digital converter may be internal to the processor 24 or 
provided as an external module. As will be appreciated, the sensor mesh 22 is intended 
to include combinations of sensors and sensor types, which can be used over the whole 
or part of the deformable surface 20. 

15 

A positional or environmental sensor system 28 can also be supported by 
device 10. Various sensor modes can be supported, including absolute or relative 
positional information as determined by gyroscopic sensors, accelerometers, or 
acoustic or infrared ranging techniques. Environmental sensors, including conventional 

20 light, image, thermal, electromagnetic, vibratory, or acoustic sensors can also be 
present. Depending on the desired application, even costly environmental or positional 
sensors such as those incorporating differential GPS positioning, image analysis or 
recognition, acoustic or voice identification, or differential thermal sensors can be used 
as morphemic input. Such morphemic input, taken in conjunction with morphemic input 

25 as detected by sensor mesh 22, can enhance precision and flexibility of a user's control 
of device 10. 

As illustrated, both sensor system 28 and sensor mesh 22 are connected to the 
processor 24 and associated memory 26. The processor 24 and memory 26 are 
30 typically mounted within the deformable surface 20, by either direct attachment to the 
deformable surface 20 or by attachment to a hard casing positioned within the 

11 
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deformable surface 20. Conventional CISC or RISC processors can be used in the 
illustrated embodiment, with low power processors such as the Signetics; 87c752 or 
87c751 f Motorola 68HC11 or 68582, or ARM 710 being preferred. If convenient, 
coprocessors such as analog to digital converters or digital signal processors can be 
5 used alone or in conjunction with a main processor. Conventional flash, static, or 
dynamic RAM can used in the present invention, although for certain applications 
higher cost embedded DRAM may also be used. In some storage intensive 
applications, memory 26 can include additional harddisk storage, either located within 
the device 10 or available through an external connection. As will be appreciated, for 
10 many applications use of optional external communications can at least partially 
supplant use of internal processors and memory (except for that necessary to support 
required sensor or communication buffering and signalling). 

The present invention optionally supports communications with an external 
15 computer system 40 using its internal communications system 32 and associated 
transceiver 34. The external computer system 40 also includes a transceiver 42, a 
personal computer or workstation 44, and is connected to a local or wide area network 
computer system 46. The transceivers 34 and 42 can support various communication 
protocols and designs, including use of a serial tethered line 36 (using, for example the 
20 RS-232C interface protocols), use of infrared signals 38 adhering to widely utilized 
IROA communication standards, or use of radiofrequency signal 37 (which can be, for 
example, a cellular telephone, 900MHz radio, or digital PCS telephonic 
communications). Alternative communication standards, or even alternative 
communication carriers such as those based on optical or acoustic techniques, can of 
25 course be employed. 

As will be appreciated, in addition to direct communication with external 
computer system 40, the device 10 can be directly or indirectly maintained in 
continuous or intermittent communication with a number of suitably equipped electronic 
30 devices, including a tablet computer 110, or even a physically manipulatable portable 
computer 11 similar in design and function to device 10. Communication can be direct 
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to a target device, or through an intermediary retransmitter such as computer system 
40. Other possible communication targets include automation control systems , security 
authorization units, personal digital assistants, notebook computers, or any other 
suitably equipped electronic system. 

5 

Results of communications with external devices, presentation of device stored 
information, or device status updates can all be provided to a user through 
processor 24 controlled update of feedback module 33. Feedback to a user can be 
primarily visual, such as can occur in conjunction with visual display 30. Generally, the 

10 display 30 can be a conventional passive or active matrix liquid crystal display, although 
use of more sophisticated (and expensive) displays based on various electrooptical or 
micromechanicai techniques can of course be used. In addition, for certain devices a 
non-imaging display such as may be formed by a small number of status lights (e.g. red 
or green LEDs), or localized or distributed chromatic changes (in conjunction with a 

15 deformable surface 22 constructed with suitable electrochromic materials) may be ail 
that is necessary for visual feedback to the user. 

In some embodiments of the invention, visual output through display 30 may be 
augmented (or even replaced) with a non-visual display 31. The non-visual display 31 

20 can include tactile displays based on internal actuators, auditory feedback, or even 
displays based on conformal changes in device appearance. For example, one possible 
feedback display is based on internal auditory speakers (emitting a range of sounds 
from simple "beeps" to well formed speech, depending on available processor speed 
and functionality) for providing user feedback. As will be appreciated, non-visual display 

25 31 and its associated actuators or electronics can support alternative feedback modes, 
including, for example, force feedback to a user through internal actuators, tactile based 
feedback (e.g with multiple surface projections for presentation of Braille or other 
conventional tactile user interface), modifications to the surface texture of the device, 
or any other conventional mechanism for supplying information to a user. 

30 
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To better appreciate operation of the present invention, some selected modes of 
physical manipulation of device 10 are schematically presented in Figure 1-.- As seen in 
Figure 1, device 10 can be translationally moved in three spatial dimensions, as 
illustrated with reference to orthogonal force arrows 50, 51, and 52. In addition to 
5 transiational motion, the device 10 can be rotationally moved in any or all three spatial 
dimensions, as represented by arrows 53, 54, and 55. By use of sensor system 28 
(alone or in combination with communication system 32), the relative or absolute 
position and orientation in three dimensions can be determined. 

10 In addition to determining spatial position and orientation through use of sensors 

28, the device 10 can optionally use sensor mesh 22 to measure and localize transient 
or continuing force application, with force vectors and related timing information being 
determined and interpreted. A few possible force actions (deformation modes) are 
schematically illustrated in Figure 1, with arrows 60 and 61 illustrating depression of 

15 surface 20 (with the combination representing a squeeze), arrows 62 and 63 illustrating 
sliding or rubbing deformation (with the combination representing a twist), and the 
combination of sliding arrows 65 and 66, and outward pull 67, together representing a 
pinch and outward pull. The strength of the applied force can be measured (e.g. a hard 
or soft squeeze is differentiated), its spatial expanse found (e.g. to differentiate between 

20 poking with a fingertip or a thumb pad), and timing determined (e.g. to differentiate 
between a quick or a slow depression of the surface). The deformation so caused may 
be either permanent or transitory. 

As will be appreciated by those skilled in the art, each of the foregoing force 
25 actions represented by arrows can be considered a senseme. Some temporally 
distinguishable sensemes (or combinations of sensemes such as the foregoing 
discussed pinch/pull combination) further represent morphemes used as a basis for a 
morphemic grammar in accordance with the present invention. All of the following 
described morphemes can be modified by numerous variations in applied pressure, 
30 force utilized, appendages used, body parts, or extraneous intermediary objects used to 
apply force. In addition, timing of various objects (whether quick, slow, or alternately 
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quick and slow) can modify interpretation of a morpheme. For example, if "squeeze" is 
taken as a typical morpheme, one can appreciate various squeeze operations such as 
quick squeeze, slow squeeze, hard squeeze, soft squeeze, narrow squeeze, wide 
squeeze, squeeze between hands, squeeze- between one hand and a user's chest or 

5 head, squeeze between one hand and a table or wall, a squeeze made between two 
pens or two books, or even a squeeze between a user's tongue and the roof of the 
mouth. For purposes of the present invention, all such squeeze morphemes would be 
considered members of the "squeeze" class, with individual variations acting as 
possible modifiers or selected cases, just as a "house cat" might be considered a 

10 particular member of the class of "felines", which more generally embraces lions, tigers, 
and bobcats. 

To aid in understanding the diversity of contemplated physical manipulation 
morphemes, Figure 2 illustrates selected morphemes arranged by increasing plasticity 
15 of the device required to enable morpheme utilization, and by increasing complexity of 
available senseme tuples required to form or interpret a morpheme applied to a 
y3 particular class of device. Beginning with the least plastic device and the simplest 

: ; senseme set used to compose a morpheme, a definition of a possible physical 

H manipulation and typical function invoked by that manipulation of a device similar (but 

m 20 of course possibly more complex) to that described in conjunction with Figure 1 is 
US presented: 

DEPRESS 

Definition: Indenting one or more subregions of the device through the 
: ■ 25 application of pressure. 

Example: As seen in Figure 3, consider a device 122 having a display 123. The 
device 122 supports a graphical illustration software application which allows users to 
position geometric shapes or user defined graphical objects. The device can have four 
pads 124 placed around it, one on each side. By deforming a particular side, the user 
30 indicates a desire to "nudge" the currently selected geometric shape 125 away from 
that side to a new position 126. 
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SQUEEZE 

Definition: To deform one or more subregions of a device through the application 
of vectors of force, some components of which are directed towards each other, which 
5 compress the morphological structure of the device. This is a special case of 
depression. 

Example: As illustrated in Figure 4, consider a device 132 which can display one 
or more documents in either iconic (closed) or textual display (open) modes. By 
selecting an open document 135, and then squeezing a deformable edge 134 of the 
io device 132, the user indicates a desire to make the document "smaller*, in this case to 
iconize it as icon 136. 

FOLD 

Definition: To deform a second subregion by bending the first subregion 
is such that it partially or completely overlaps the second subregion. Further deformations 
can be applied to that new morphological structure on other subregions. 

Example: As illustrated in Figure 5, consider a device 142 which can display 
documents. Suppose this device 142 has been augmented such that a deformable 
horizontal "flap" 144 on a top edge of the device 142 can be folded to partially obscure 
20 a display 143. When the user makes this folding gesture, the user indicates a desire to 
password-protect ("hide") the currently displayed document. 

CURL 

Definition: Deforming one or more subregions of the device by spirally 
25 configuring subregions relative to each other in a cylindrical or circular morphological 
structure. 

Example: As illustrated in Figure 6, consider a device 150 which can display 
documents in various languages (English, French, etc.). When the user takes such a 
device 150, curls in a direction indicated by arrow 157 into a tube, and then uncurls it, 
30 this "abra-cadabra" gesture tells the device 150 to display the current document in a 
different language. 
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STRETCH 

Definition: Deforming one or more subregtons of a device through the application 
of vectors of forces, some components of which are directed away from each other, the 
vectors being applied at opposite ends of the device. Example: As illustrated in Figure 
7, consider a device 160 with a graphical software application which allows users to 
manipulate geometric shapes. By stretching the device 160, the user indicates a desire 
to "resize" or "rescale" the currently displayed shape165 to a larger size 166, the 
amount of resizing being a function of the amount of deformation. Note that 
SQUEEZING can indicate resizing the currently displayed to a smaller size. 

PINCH 

Definition: Manipulating one or more subregions by applying vectors of forces, 
aligned directly towards each other, on opposite sides of the affected subregion(s). This 
is typically, but not exclusively, accomplished using two-finger tactile force. PINCH is a 
special case of SQUEEZE. 

Example: As illustrated in Figure 8, consider a device 170 which can copy 
documents. By performing a "pinching" action 175, the user indicates that they wish the 
next set of copies to be issued in stapled form. 

DOGEAR 

Definition: Deforming a second subregion by folding a first subregion at a logical 
comer or edge of the second subregion, indicating a marker or location point (e.g. a 
bookmark) to be later referenced. DOGEAR is a special case of FOLD. 

Example: As illustrated in Figure 9, consider a device 180 which displays a 
subset of the pages from a multi-page document. By "dogearing" the upper right comer 
185 of device 180, the user indicates that they wish a bookmark associated with the 
currently displayed page or pages. 
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TWIST 

Definition: Deforming one or more subregions of the device through the 
application of two opposing rotational forces offset from each other by a non-zero 
difference about some central axis. 

Example: As illustrated in Figure 10, consider a device 190 which, over time, 
becomes degraded in some aspect of its performance (its disk becomes fragmented, its 
memory needs garbage-collection, etc.)- By performing a "TWIST" gesture 195, the 
user indicates that they wish the device to "wring itself out", performing, for example, 
garbage collection. 

RELIEF-MAP 

Definition: Deforming one or more subregions of the device by raising and or 
lowering them by either spatial transformation or the addition/removal of material. 

Example: As illustrated in Figure 11, consider a device 200 which can display 
documents in either one- or two* page format. When the user "scores" the device by 
making a vertical indentation about the center axis of the device while it displays a 
single page 206, the device 200 interprets the morpheme to request display of 
documents in a two-page format as pages 207 and 208. 

RIP 

Definition: Deforming one or more subregions of the device by introducing a 
spatial discontinuity, by applying vectors of force to partially or totally disconnect these 
subregions from the device. 

Example: As illustrated in Figure 12, consider a device 210 which can copy some 
or all of its information. When the user performs the "ripping" gesture 215, removing 
one or more subregions, the device 210 copies its currently selected data set onto 
those subregions 216 and 217. 
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PERFORATE 

Definition: Deforming one or more subregions of the device by- means of 
introducing a change in the spatial connectivity of the subregions such that a hole is 
introduced (either temporarily or permanently) in the device. 

Example: As illustrated in Figure 13, consider a device 220 which is used to 
route messages between various parts of a device network, and which displays this 
functionality to the user by means of lines 222 representing message pathways. When 
the user perforates the device 220 with a finger or object 224 to introduce a hole in one 
of these pathways 222, the system stops routing messages along that pathway. 

SIMILARITY 

Definition: Deformation of one or more subregions of a device previously 
configured to represent some other predefined object. Typically the device acts in a 
manner consistent with the behavior of the real-world object when it is manipulated in 
this manner. 

Example: As illustrated in Figure 14, consider a device 230 which contains text- 
to-speech and audio input capability, and which presents itself to the user in an 
anatomically accurate shape of a human head. When the user opens lips 232 on the 
human head, internal sensors detect opening of the lips and activate text-to-speech 
capability. 

3D MAP 

Definition: The morphing of a device that can be molded around an external 
object to permit a sensor mesh to determine simultaneously size and shape of the 
external object. The range of possible external objects is large but is limited by the size 
of the device's solid inner-housing and the volume of the moldable material in the outer 
housing. In this system the device has the ability to accurately sense the amount of 
material from its inner surface to the outer edge of the moldable material (example: via 
ultrasonic sounding, similar to sonar in water) thus determining an accurate electronic 
model for the shape of the molded enclosure. 
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Example: As illustrated in Figure 15, by pressing a device attached moldable 
material 244 around the surface of an external object (e.g. a cog 242) a device 240 can 
automatically generate a CAD model of that object and store it in its memory. 

5 MIMICRY 

Definition: Deforming one or more subregions of the device such that the 
resultant morphological structure resembles a known real-world object and through this 
association of subregions, the device acts in a manner consistent with the object it 
resembles. 

to Example: As illustrated in Figure 16, consider a device 250 containing a 

computer which contains text-to-speech and audio input capability, and which presents 
itself to the user as a misshapen blob having the consistency and plasticity of moldable 
putty or clay. When the user performs the "MIMICRY" action by molding part of the 
device 250 to resemble an ear, the audio input capability is activated. 

15 

In addition to morphemes based on physical manipulation, various morphemes 
based on varying degrees of relative or absolute spatial positioning are contemplated to 
be useful in practice of the present invention. To aid in understanding the diversity of 
contemplated spatial morphemes, Figure 17 illustrates selected spatial morphemes 

20 arranged by increasing knowledge of spatial position required to enable morpheme 
utilization, and by increasing complexity of available senseme tuples required to form or 
interpret a morpheme applied to a particular class of device. Beginning with a device 
having only rudimentary relative positioning functionality for supporting simple spatial 
sensemes, and ending with a device absolutely positionable to within centimeters 

25 anywhere on Earth, a definition of a possible spatial manipulation and typical function 
invoked by that manipulation of a device similar (but of course possibly more complex) 
to that described in conjunction with Figure 1 is presented: 

TRANSLATE (relative to device) 
30 Definition: The linear movement of a device's center of mass from one position in 

space to another. 
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Example: Used to substitute for mouse controlled graphical "sliders 1 in 
conventional graphical user interfaces. When it is only physically possible to display a 
small amount of list, large listings can still be searched by "scrolling" a display window 
in response to the TRANSLATE morpheme. 

SHAKE 

Definition: Spatially translating all subregions of a device by repeated movement 
in opposing directions, such that the net translation is negligible. 

Example: As illustrated in Figure 18, consider a device 260 which is used as a 
calculating device. When the user performs the "SHAKE" gesture, the device 260 
clears its accumulator. 

REVOLVE 

Definition: Rotating all subregions of a device by rotating the subregions about a 
point internal to the device, about any arbitrary plane. 

Example: As illustrated in Figure 19, consider a device 270 which displays an 
imaged slice of volumetric data, such as medical data from a series of CAT scans. By 
rotating the device about a center point 272 internal to the device 270 to a new position 
274, the plane specifying the imaged slice is changed accordingly. 

TILT 

Definition: Rotating one or more subregions of a device by rotating 
the subregions such that one or more components of the rotary force are in the 
direction of gravity and the amount of rotation is between about -180 degrees and +180 
degrees. 

Example: Consider a device which displays frames from an animation sequence 
on the side facing the user. As the device is tilted away, the speed of the animation 
increases - as its tilted towards the user, the speed of the animation decreases, 
analogous to operation of a gas pedal. 
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FLICK 

Definition: A forwards TILT immediately followed by an opposing- backwards 

TILT. 

Example: As illustrated in Figure 20, consider a device 280 which can transmit 
some subset of its data to another device. When the user performs the "FLICK" gesture 
by quickly tilting in the direction of arrow 282, followed by a reverse tilt along arrow 282, 
the device 280 performs this transmission, towards the device (not shown) pointed to by 
the ray of the gesture. 

SPIN 

Definition: Rotating one or more subregions of the device by rotating the 
subregions about a point internal to the device, such that the plane of rotation is one of 
the device's surface planes. SPIN can be considered a special case of REVOLVE. 

Example: As illustrated in Figure 21, consider a device 290 which can display a 
frame of video 295 from a video sequence. The user performs the "SPIN" gesture in a 
counter-clockwise direction, the device displays earlier frames in the sequence; when 
the gesture is performed in a clockwise direction (arrow 292), the device 290 displays a 
later frame 296 in the sequence (represented by film strip 294). 

ORIENT 

Definition: Rotating one or more subregions of the device by rotating the 
subregions about the center of the device, such that the plane of rotation is one of the 
device's surface planes, and the amount of the rotation is a multiple of 90 degrees (i.e. 
to rotate the device between cardinal compass points). Orient can be considered a 
special case of SPIN, which in turn is a special case of REVOLVE. 

Example: As illustrated in Figure 22, consider a device 300 which can display a 
document in either 1 page, 2-page, or 4-page format ("1-up", "2-up", or "4-up"). When 
the user performs the orient gesture in a clockwise direction (arrow 302), the device 300 
increases the number of document pages it is displaying from one page 305 to two 
pages 306 and 307. Further orient gestures would increase the number of displayed 
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pages. When performed in a counter-clockwise direction, the device 300 decreases the 
number of pages it is displaying. ; 

FACE 

Definition: Manipulating one or more subregions of the device such that a first 
set of subregions is no longer bottom-most and a second distinct set of subregions now 
assumes the first subregions former position. 

Example: As illustrated in Figure 23, consider a device 310 which displays 
documents, and which allows users to edit such documents. Consider further the case 
in which the device presents itself to the user in the form of a cube, in which 6 different 
documents are displayed on the 6 different faces. When the user performs the "FACE" 
gesture by making a particular face top-most, the document which Is now on the top- 
most face becomes editable by the user, while the document which is no longer top- 
most is no longer editable. 

LIFT 

Definition: The movement of a device's center of mass in a direction opposite to 
the current gravitation force acting on the device. 

Example: Commanding the device to display the computer's file system at a 
position one level higher in the hierarchy. 

PAN 

Definition: The application of a TRANSLATION to a device such that it is moved 
parallel to the front of the user's body, at a substantially constant height 

Example: Viewing a spread-sheet on a device with a display so small only one 
cell can be shown. By PANing the device, the contents of the current row can be shown 
in sequence depending on the rate or amount of PAN. However, if the device were 
rotated away from its current orientation during the PAN, a new row would be chosen. 
The selection of the row could be dependent on the deviation from the original 
orientation. 
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PUSH-PULL 

Definition: Manipulating one or more subregions of the device by spatially 
translating them such that they are moved along a line of projection from the center of 
the device to the vertical axis of the user's body. 

Example: Consider a device with audio output capability. As the device is 
"pushed" further away from the body, its audio output level increases. When its "pulled" 
towards the body, its level decreases. 

WHACK 

Definition: The application of an accelerative or de-accelerative force to one or 
more subregions of a device such that said subregions contact or are contacted by an 
external object, causing an equal and opposite countering force. 

Example: As illustrated in Figure 24, consider a device 320 which can perform 
long and unpredictable database searches. When the user performs the WHACK 
gesture (e.g. upon a table 322), the current search is aborted. 

ORIENT (relative to environment) 

Definition: Manipulating two subregions of the device such that the line drawn 
between the centers of those two subregions alters its orientation with respect to the 
surrounding environment. 

Example: Displaying a CAD drawing of a machine-part in 3D on the display of a 
mobile device. As the orientation of the device changes, so too does the viewing angle 
and position of the rendered image. 

ORBIT 

Definition: Rotating one or more subregions by rotating the subregions and/or the 
center of mass of the device about some point exterior to the physical boundaries of the 
device, about any arbitrary axis. 

Example: As illustrated in Figure 25, consider a device 330 which can search a 
network database, such as the World Wide Web, for information. When the user 
performs the "orbit" gesture, such a search is initiated. The radius 332 of the 
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rotation 334 specifies the breadth of the search - wider circles specify a wider search. 
The speed of the gesture specifies the time limit imposed on the search • the quicker 
the gesture, the more cursory the search. 

ORBIT RELATIVE TO USER 

Definition: Rotating one or more subregions by rotating the subregions and/or the 
center of mass of the device about some point exterior to the physical boundaries of the 
device, where said point is proximal to a body feature of the user. This is a special case 
of ORBIT. 

Example: As illustrated in Figure 26, consider a device 340 which can perform 
audio output. By performing the ORBIT gesture (in direction indicated by arrow 344) 
about the user's ear 345, the audio output is activated. 

MOVEMENT IN ROOM 

Definition: The local detection of a device's 3D position relative to reference 
points found within an enclosing room. Differences in the measured position are used to 
trigger actions. 

Example: A virtual filing system that allows you to save and restore files based 
on the devices current position in the room. To save a file you might think carefully 
about the contents of the file and then walk to the position in the room that might be 
most easily associated with it. When retrieving the file you would use the same thought 
processes and go back to the position you had associated with the file. On doing so the 
files associated with that position would be displayed and you would then be able to 
select the file you were looking for. The system is useful because the human mind is 
very good at remembering information that is spatially organized rather than in some 
abstract information data structure. 

MOVEMENT BETWEEN WIDELY SEPARATED SITES 

Definition: Manipulating one or subregions of the device such that the detected 
absolute spatial position of those subregions is changed. 
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Example: Consider a device which can display information from a database of 
client information. When the device is moved to a different client site, the device 
automatically updates its display to display information for the nearest client site. 

5 In addition to morphemes based on physical manipulation or spatial positioning, 

various morphemes based on sensed environmental conditions are contemplated to be 
useful in practice of the present invention. To aid in understanding the diversity of 
contemplated environmental morphemes, Figure 27 illustrates selected environmental 
morphemes loosely arranged in order of increasing sensor complexity needed in some 
to commonly sensed environmental categories. For each of- the proffered categories, 
some selected sensing systems supportable by a device such as that described in 
conjunction with Figure 1 are presented: 

LIGHT 

15 Definition: Manipulating one or more subregions of the device such that the 

amount of light falling upon those subregions changes. 

Example: Consider a device used in a lecture hall to take notes. When the room 
lights are turned on, the light sensors detect this and turn down the backlight to 
conserve energy. When the room lights are turned off (e.g. during a slide show) the 
20 light sensors detect this and turn up the backlight to increase viewability. 

Light sensors can range from threshoided binary light detectors, to light pattern 
detectors, to full imaging systems. Advanced techniques can include image analysis 
and recognition to identify objects or persons. 

25 HEAT 

Definition: Manipulating one or more subregions of the device such that the 
amount of heat applied to those subregions changes. 

Example: Consider a portable computer which has a stylus for entering text. By 
looking at the heat profile along the back surface of the computer, the computer can 
30 detect whether it is being held with the left hand, the right hand, both hands, or neither 
hand, and update its interface accordingly. 
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Thermal (heat) sensors can range from simple temperature sensors to 
sophisticated differential thermal mappers and thermal imagers. 

ELECTROMAGNETIC 

Definition: Manipulating one or more subregions of the device such that the 
electromagnetic spectrum applied to those subregions changes. 

Example: By analyzing the radio spectrum, the device can derive estimates as to 
its absolute spatial position, and use that to alter its functionality. 

Electromagnetic detection can include magnetic compasses, radio detection, or 
GPS signal detection. More advanced techniques can include electromagnetic 
spectrum analysis and interpretation, such as roughly determining location based on 
available radio signals. 

VIBRATE 

Definition: Manipulating one or more subregions of the device by vibration. 

Example: Consider a device which displays textual information. When the user 
takes the device on the bus, the ambient vibration level sensed by the device changes, 
and the device increases the size of the displayed text to help the user compensate. 

This class of environmental morphemes can include detection of intermittent 
contacts, low frequency rumblings, or acoustic level detection. More advanced 
techniques requiring greater processor power include maximum frequency 
identification, spectral analysis of acoustic frequencies (enabling the device to 
distinguish background environmental noises from speech, for example), or even 
speech based identification of persons in the vicinity of a device. 

In addition to morphemes based on physical manipulation, spatial position, or 
sensed environmental factors, various morphemes based on cooperation between 
multiple interacting devices are contemplated to be useful in practice of the present 
invention. To aid in understanding the diversity of contemplated spatial morphemes, 
Figure 28 illustrates muftidevice morphemes arranged by an increasing level of possible 
physical contact, and by increasing complexity of available senseme tuples required to 
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form or interpret a morpheme applied to a particular class of device. Beginning with a 
device having only rudimentary edge deformation functionality for supporting simple 
spatial sensemes, and ending with complex deformable or embeddable devices that 
can be wrapped about each other, a definition of a possible multidevice manipulation 
and typical function invoked by that multidevice manipulation of devices similar (but of 
course possibly more complex) to that described in conjunction with Figure 1 is 
presented: 

TOUCH 

Definition: To move one or more subregions of a device such that they enter 
physical contact with a subregion of a second device, in any alignment and to any 
extent. Or, to take two devices so aligned and remove that alignment. 

Example: Consider two portable computers 350 and 351, the first of which 
contains a database, and the second of which contains an IRDA port. When the user 
touches the first computer 350 to the second computer 351 , the database is transmitted 
via the second computer's port. 

MATCH 

Definition: To move one or more subregions of a device such that they enter 
physical contact with one or more subregions of a second device, with the subregion(s) 
of the first device and the subregion(s) of the second device being aligned along one or 
more edges. Or, to take two devices so aligned and remove that alignment. 

Example: As illustrated in Figure 30, consider, multiple devices 360, 361, and 
362, which contain different versions of the same basic database. When the user 
MATCHes the first device 360 to the second device 361 , followed by matching the third 
device 362 to the matched first and second devices, their databases are reconciled 
(synchronized). 

STACK 

Definition: To move one or more subregions of a device such that they enter 
physical contact with a subregion of a second device, such that the first device is now 
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located above, but physically adjacent to, the second device. Or, to take two devices so 
aligned and remove (i.e. unstack) that alignment. 

Example: As illustrated in Figure 31, consider a set of devices 370, 371, and 
372, each of which is displaying a frame of video from a longer video sequence. When 
5 the devices are stacked, the ordering of the stacking specifies an order for the video 
editing, and a single composite video is now produced. 

TILE 

10 Definition: To move one or more subregions of a device such that they physically 

contact with a subregion of a second device, such that the first device and second 
device now form a single seamless spatial unit or to take two devices so aligned and 
remove that alignment. TILE is a special case of MATCH. 

Example: As illustrated in Figure 32, consider a set of devices 380, 381, 382, 

15 383, each of which can independently display a portion of a large photograph. When 
the devices are tiled, each device displays the portion of the photograph appropriate to 
its current relative position in the tiled grid. 

RELATIVELY ALIGN 

20 Definition: To move one or more subregions of a device such that they engage in 

a particular spatial relation to one or more other devices, where said devices are not 
touching. 

Example: As illustrated in Figure 33, consider a set of devices 390, 391, 392, 
and 393 which are displaying a multi-page document. Whichever device is presently 

25 placed at the far left (device 390) displays the table of contents, whichever one is 
presently placed at the far right (device 393) displays the index, and the others display 
pages according to their respective locations. As different devices can have different 
display capabilities, moving them about can alter the document display. For example, if 
only one of the devices has a color display, when it is moved from second position to 

30 third position then (a) the device which was in third position, displaying page #2, now 
displays page #1 , and (b) the color display, which was displaying page #1, now displays 
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page #2 in color. 

* 

WRAPPING/EMBEDDING 

Definition: Manipulating one or more subregions of a device such that these 
5 subregions spatially occlude or are spatially occluded by some portion of the second 
device. 

Example: As illustrated in Figure 34, consider a first device 400 which contains 
the infrastructure to support the filtering of email. Consider a second set of devices 401 
and 402 which implement particular email filters. When the user physically embeds 
10 device 401 (or 402) into the first device 400, by which the first device 400 now wraps 
the second device 401 (or 402), the particular email filter supported by the second 

0 device is activated. 

As those skilled in the art will appreciate, combinations of any of the foregoing 

15 described morphemes based on physical manipulation, spatial position, environmental 

53 conditions, or multiple interacting devices can be extended by participation in a 

1 morphemic "sentence". A sentence is defined as a sequence of one or more temporally 
ft disjoint morphemes. Typically, between about 1/1 0th of a second and 2-3 seconds 
fli suffices to distinguish morphemes within a sentence. Of course, in some circumstances 
lZ 20 and indefinite time period may elapse. The sentence level allows definition of a physical 
O manipulatory grammar by appropriate choice of a morpheme sequence, and corollary 

rules governing, for example, use of active (verb like) morphemes, naming (noun) 
morphemes, or connectors. Just as the position and relation of words in a sentence 
define the sentence's meaning (e.g. "horse chestnut" is not the same as "chestnut 

25 horse"), similarly the position and relation of morphemes in a manipulatory sentence 
define the sentence's meaning. For example, in a communication mode a FLICK 
followed by a WHACK could mean "transfer data and erase the local copy", while a 
WHACK followed by a FUCK means "power on the device and transfer data". In other 
settings, a FLICK or a WHACK could mean something entirely different To better 

30 understand construction of morphemic sentences, the following examples are 
discussed: 
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DATA TRANSFER SENTENCE 

Consider a device which can transmit some or all of its information to another 
device. Furthermore, this transmission can be done either unencrypted, or encrypted (to 
increase security). Furthermore, this transmission of a text+graphics document can 
either include the graphics, or omit them (to save time). Suppose that the user wishes 
to perform the command Transmit the information in document A, encrypted, omitting 
graphics, to machine B". Then a gestural sequence (morphemic sentence) to support 
this could be: 

DEPRESS - the user presses on a displayed representation of A, indicating that 
A is to be selected for an upcoming operation 

FLICK- the user flicks the device in the direction of device B, indicating that the 
operation is a transmission to B 

FOLD - the user folds the top quarter of the device over the lower three-quarters, 
indicating that the transmission is to be encrypted. 

TWIST - the user twists the device about its central axis, indicating that the data 
is to be "wrung out", i.e. the graphics are to be omitted 

SQUEEZE - the user squeezes the device, indicating that they are confirming 
that the operation is to proceed. 

Note that none of these gestures, in isolation, performs an act - the "ensemble" of 
temporally separated morphemes must be interpreted in order to form the complete 
action. 

DRAWING MODIFICATION SENTENCE 

Consider a device which displays geometric shapes for user manipulation. 
Furthermore, one of the manipulations supported is to resize (or rescale) a shape. 
Furthermore, suppose that this resizing can be done either aliased G a 99Y) or anti " 
aliased (edges are smoothed). Suppose that the user wishes to perform the command 
u Resize shape A by 120%, about the X axis only, using anti-aliasing". Then a 
morphemic sentence to support this could be: 
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DEPRESS- the user presses on a displayed representation of A, indicating that 
A is to be selected for an upcoming operation 

STRETCH - the user stretches some portion of the device, indicating that the 
operation is to be a resize. When the user starts stretching, a portion of the status 
display displays "100". The user continues to stretch until the status display reads 
"120" 

RELIEF-MAP - the user "scores" the display by making a horizontal line of 
depression, indicating that the operation is to take place only about the horizontal (X) 
axis. 

DEPRESS - a circular thumb stroke is made in a different area of the device, 
indicating that anti-aliasing (smoothing the edges) is to be performed. 

DATABASE PRESENTATION SENTENCE 

Consider a device which contains various personal information databases, such 
as a list of phone numbers, a list of addresses, and a calendar. Suppose that the user 
wishes the most appropriate of those databases displayed. Then a gestural sequence 
to support this could be: 

SPATIAL LOCATION - the user carries the device such that it is spatially 
proximate to either the telephone, the address book, or the refrigerator (where the 
family calendar is displayed), whichever is appropriate. 

DEPRESS - the user touches the device to activate the operation. 
The device now displays the personal information appropriate to that location. 

DATABASE RETRIEVAL SENTENCE 

To extend the foregoing example of a database presentation sentence, consider 
two computers which contain calendar databases. If the user wishes to synchronize the 
calendars, a suitable gestural sequence to support this could be: 

SQUEEZE - the user squeezes the device to activates its gestural recognition 
capabilities 

ORBIT - the user orbits the device about the surface of the other device 3 times, 
indicating a desire to only match data for the next 3 weeks. 
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MATCH - the user matches the edge of the device to the edge of the calendar, 
indicating a desire to "match" contents between the two devices. 

PRINTER/COPIER CONTROL SENTENCE 

Consider a device which can produce paper copies of documents. Suppose that 
the user wishes to teil such a device to produce a stapled, two sided copy of document 
A, enlarged to the next greater size. Then a gestural sequence to support this could be: 

DEPRESS - the user presses on a displayed representation of A, indicating that 
A is to be selected for an upcoming operation 

RIP - the user introduces a spatial discontinuity into a portion of the device, 
indicating that the upcoming operation is to be a copy ("carrying away" some of the 
data). 

PINCH - the user pinches the upper left corner of the device, indicating that the 
copies are to be stapled. 

SQUEEZE - the user presses on the front and back of the device, indicating that 

the copy is to be two-sided. 

STRETCH - the user stretches the device, indicating that the copy is to be an 
enlargement to the next greater size. 

FACE - the device typically has its paper emitter on the bottom, preventing users 
from making copies accidentally. By facing the device such that the emitter is on the 
side, the copying operation is initiated. 

LIGHT BASED CONTROL SENTENCE 

Consider a device which can display documents. Suppose the user is using the 
document while seated on a train, and wishes the document to display itself with a 
backlight when the train enters a tunnel, and wishes the document to display itself in a 
larger font when the train rumbles over rough tracks. Then a gestural sequence to 
support this could be 

SQUEEZE - the user squeezes on the device, indicating that a loss of light is to 
be compensated for by a backlight. 
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LIGHT - as the train enters a tunnel, the LIGHT gesture is made, and the device 
tumson the backlight. 

WHACK - the user briskly raps the device against the palm of their hand, 
indicating that their desired preference for adjusting hard-to-read documents is to 
increase the font size. 

VIBRATE - as the train goes over a bridge, the vibratory gesture is sensed. 
Because of the position of this VIBRATE morpheme (after the preceding WHACK 
gestae) in this morphemic sentence, the device now increases the font size on the 
disptayed text. 

LIGHT - the user puts the device into his suitcase, making the LIGHT gesture. In 
this context (with no SQUEEZE before it), the LIGHT gesture causes the device to 
power-off its display. 

To better appreciate utility and construction of devices in accordance with the 
present invention, several examples of devices are now described: 

PORTABLE COMPUTER WITH SQUEEZE AND TILT CONTROL 

A handheld portable computer 500 (e.g. a 3Com ® PalmPilot ® ) capable of 
being fitted with deformable, pressure sensitive edging 504 is schematically illustrated 
in Rgures 35 and 36. The computer 500 supports a name and address software 
application, providing a user viewable name-address entry field on display 503. In this 
embodiment, a user can squeeze the deformable, pressure sensitive edging 504 
(squeeze arrows 507) of the computer 500. In response, the name and address 
software application causes the display 503 to animate by slowly incrementing 
(scroling) through the name list from "A" towards "Z* entries. When the user squeezes 
edging 504 again, the software application stops the scrolling animation. Scrolling 
functionality is further enhanced by the use of a tilt sensor, which allows the computer's 
behavior to mimic conventional rotatable address books. If the computer 500 is tilted 
away from the 45 degree angle at which someone might typically hold it, the scrolling 
rate is increased. In this application, the closer the computer 500 was tilted towards the 
user (as indicated by arrow 506 of Fig. 36), the faster the scroll rate toward "Z*. 
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However, if a user tilted the computer 500 back past the neutral 45 degree position (as 
indicated by arrow 506 in Figure 36), the animation would move backwards with a 
velocity related to the magnitude of tilt. In this way it was possible for a user to search 
for items in a long list in a very natural way, while only using one hand. 

5 

In an alternative mode schematically illustrated in Figure 37, scrolling speed can 
be completely controlled by pressure. The greater the squeeze pressure (arrows 537), 
the faster the list scrolls. Release of the applied pressure causes the scrolling to halt. In 
this alternative user interface strategy, application tilt (as indicated by orthogonal tilt 

10 arrows 530 and 532) could be used to change the direction of the scrolling through the 
list, allowing a user to search portions of a large two dimensional data set 
(schematically illustrated as a data plane 520) that is not entirely visible either 
horizontally or vertically on display 503. By simply tilting the display 503 of computer 
500 as if it were a window through which the data plane 520 can be viewed, any 

15 particular portion of the data plane (for example, data subset 524) can be viewed. As 
will appreciated, in both the foregoing modes the speed of scrolling, the specific neutral 
tilt angle, and required pressures to initiate scrolling changes can be adjusted to fit a 
particular user. 

20 PORTABLE COMPUTER WITH HANDEDNESS DETECTION 

Pressure sensors have been added to augment a conventional keyboard 551 
enabled user interface to a hand holdable Windows ® CE class computer 550 (i.e. a 
Cassio ® Cassiopia ®) schematically illustrated in Figures 38 and 39. In this 

25 embodiment, user handedness was determined by using pressure sensors positioned 
on a right back-edge and a left back-edge of computer 550. User studies have found 
that the difference in pressure between the right and left side gave a direct indication of 
handedness of a user. As respectively illustrated in Figures 38 and 39, handedness 
was used to justify formatted text 554 to the left (Fig. 38) or right (Fig. 39), thereby 

30 allowing more space 555 on display 553 for an electronic annotation pen to be used to 
mark-up the text. 
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For both the embodiment of the invention illustrated by Figures 35-37, and the 
foregoing embodiment illustrated in Figures 38 and 39, material deformation of a 
spongy, elastic, or otherwise deformable material must be measured. Although various 

5 techniques can be used to measure material deformation, including those based on 
imaging or fluid volumetric changes, one particularly useful technique is based on the 
use of pressure transducers. Commercially available sensors measure pressure 
(indicative of material deformation) by converting a pressure change to a change in 
electrical characteristics. For example, inexpensive sensors that change resistance in 

io response to pressure can be obtained in a variety of shapes and sizes, including paper 
thin sensors, and easily bendable sensor strips. Sensors of this kind can be customized 
to any particular shape or form that a gestural Ul might require. The change in 
resistance is usually linearly related to pressure, with the sensor generally being placed 
in a potential divider network to model the pressure as a change in potential. For a 

is practical circuit the resulting signal needs to be amplified, buffered and translated such 
that the change in value from minimum pressure to maximum pressure spans a useful 
range. The modified signal can now be fed into an analog to digital converter (ADC) to 
produce a digital representation of pressure. An 8-bit ADC can typically be used for 
most applications, however, if greater sensitivity to pressure changes is required, a 

20 higher resolution ADC (e.g 16-bit ADC) can be used. As will be appreciated, the ADC 
could be memory mapped into the processor's address space as a peripheral, or 
alternatively supplied as a retrofitted pressure interface to existing computers capable 
of benefiting from this system. Since an RS232 connection is an interface that is almost 
universally available on portable computers, one strategy is to have the parallel output 

25 of the ADC converted into a serial RS232 frame using a serializer such as a UART and 
then level-shift and buffer the signal as specified by the RS232 standard. At the 
computer end of the serial interface, another level-shifter and UART, the output of 
which is readable by the processor, performs the serial-to-parallel conversion. 

30 As described with reference to Figure 40, in realizing a working system a 

microcontroller 564 (a Signetics 87c752) with an ADC built in to combine many of the 
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I/O tasks in a single chip can be used in combination with level shifter 566 (a 
MAX3223). This approach has the advantage that intelligent processing of the input 
signal is possible with software. Tilt measurement was provided by a tilt sensor 567 
connected to a buffer/amp 562 to supply an analog signal to microcontroller 564. The 
5 pressure measurement can also be encoded within a protocol across the serial link. 
This particular microcontroller 564 has five ADC inputs, but by making use of eight 
digital control lines, it is possible to use only one ADC input and one buffer 
amplifier 561 , to measure up to eight pressure points with pressure sensors 565. This is 
achieved by using the control lines to select only one sensor at a time and take a 
10 reading for each using a single input to the ADC. After eight sensors have been 
selected, eight readings are acquired in memory. The design is practical because the 
microcontroller can take measurements and make analog to digital conversions at a 
rate far higher than is needed to communicate with a host computer 569. 

15 For scrolling or handedness based software applications, 16 levels were 

determined to be adequate for representing a pressure measurement. In order to have 
a high data throughput to the host computer 569, each measurement was encoded in 
one byte of an RS232 frame such that the four lowest bits were the pressure 
representation and the highest four bits were the sensor ID. Each frame of the RS232 

20 data was therefore completely self contained. Of course any protocol that limits itself to 
a specific number of devices in its address space will eventually have the problem that 
someday applications might be designed that need to reference far more devices than 
are supported. The solution used in this protocol is to reserve the sensor ID number 15 
as a special value that can extend the semantics of the encoding to include an arbitrary 

25 number of bytes representing a sensor or value. For the described software 
applications, the commonly used RS232 frame format (1 start, 8 data no parity, 1 stop 
bit at a baud rate of 9600) was chosen. 

In operation, the host computer 569 determined handedness by utilizing 
information about the current pressure exerted on two subregions - one, on the back of 

30 the device, occupying roughly the left half, and the other, also on the back of the 
device, occupying the right half. The current pressure values were converted from 
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analog to digital form, such that a digital value of 0 (zero) represents no pressure, and a 
digital value of, for example, 15 represents maximum pressure. The detection circuitry 
then proceeds: 

if (left_sensor is high AND right_sensor is high) then 
5 conclude that the user is gripping the device with both hands 

else if (left_sensor is high AND right_sensor is low) then 

conclude that the user is gripping the device with the left hand only 
else if (left_sensor is low AND right_sensor is high) then 

conclude that the user is gripping the device with the right hand only 
10 else if (left_sensor is low AND right_sensor is low) then 

conclude that the user is gripping the device with neither hand 

Also, to optimize communication, pressure values are only sent when they change. In 
order to account for jitter and error in the pressure sensors, a sensor is only considered 
15 'high' if its value is higher than some minimum threshold (e.g. "2' on the 0 to 1 5 range). 

SCANNER/PRINTER/COPIER SUPPORTING EDGE DEFORMABLE DISPLAY 

A scanner/printer/copier device 570 is schematically illustrated in Figure 41. As 
20 seen in Figure 41 (and in more detail in Figure 42) the device 570 supports a display 
574 having a deformable edge 572, with the combination roughly shaped like a piece of 
paper. In operation, a user can place a written document in the device 570 for 
scanning. An electronic version of the scanned document is displayed (i.e. as text 575 
in Figure 42) on the display 574. By outwardly pulling the deformable edge 572 as 
25 indicated by arrow 577, a user can instruct the device 570 to resize the document 
before printing or copying. Pinching opposing sides of the deformable edge 572 (arrows 
578) can further instruct the device 570 to print double sided copies. As will be 
appreciated, various other morphemes can be used to provide a simple interface for 
interacting with device 570. 

30 
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TILEABLE AND STACKABLE PORTABLE DISPLAYS 

Multiple autonomous display tiles having an onboard display controller and at 
least one surface consisting substantially of a conventional display are particularly 
useful for the practice of various aspects of the present invention. Such tiles can be 
interconnected in response to various morphemes such as TOUCH, FLICK, 
RELATIVELY ALIGN, or WHACK, or even can be operated without substantial user 
mediated morphemic input in certain situations. 

Advantageously, tile positioning can be used as an interface specifier in and of 
itself. For example, each display tile can contain a video segment in independent 
memory. Shuffling or reorganizing the tiles can allow users to physically manipulate the 
sequence of video segments to affect a physically manipulatable video editing system. 
Using the card analogy, tiles can be used to re-sequence documents, pages within a 
document, audio annotations, voice mail, or other temporal media contained within the 
tiles. The resultant sequence can then be played as a whole by using the tiled structure 
as a unit. 

For purposes of the present invention, display tile array configurations can be 
categorized as follows: 

Close-packed display tiie array 600 (Figure 43) in which tiles 602 are arranged to 
span a surface 610 in closely abutting, but not overlapping relationship, such that the 
continuous display area is maximized (i.e. seamlessly). As will be appreciated, the tiles 
could themselves form a freestanding continuous surface, or they could be disposed on 
a table or other suitable support. Each of the tiles 602 supports a display 604 sized to 
substantially cover a front surface of each tile 602. In certain embodiments, a back 
surface of each tile 602 can also support a display. Advantageously, this would allow for 
creation of free standing displays with images visible on the front and back. The 
surface 610 can be a plane, a sphere, or any arbitrary shape that permits tiling. 



39 



Attorney Docket No.: D/97403 
Express Mail No. EG 366 228 832 US 

Loose-packed display tile array 620 (each tile 622 having a display 624 as seen 
in Figure 44) in which tiles 622 are slotted into a lattice pattern (indicated by dotted lines 
625). Each tile can be considered to be situated in a regular bounded lattice slot 
(namely, the lattice slot that contains a defined center of each of the tiles) within the 
lattice where the dimensions of the bounded slot are no more than a few times the 
maximum dimension of the tile, and there is no more than one tile in any slot. Within the 
lattice regions any of the tiles can be positioned arbitratrily and still retain the same 
inter-relationship with the group. It is still possible for tiles to touch one another at the 
boundaries of two or more lattice slots, however, this is not a requirement. 

Free-format display tiles 630 (each tile 632 having a display 634 as seen in 
Figure 45) are similar to loose-packed display tiles 620 where the size and shape of 
the lattice slots may vary freely (e.g. the dimension of a lattice slot may be many times 
the maximum dimension of any of the participating tiles). The only constraint on the 
arrangement is that there must be no ambiguous relationships about the relative 
connectivity of the each of the tiles 632. That is to say, a tile that is to display the next 
piece of information to one side of another tile must be unique and not be confused with 
the task of another tile in the tiling lattice. 

3D display tiles (packable display tiles) are created by extending the three 
foregoing display tile categories. However for close packed tiling of display tiles, packed 
devices in the center of a 3D structure would not be available as the user interface. This 
may not matter as the surface of the 3D shape will expose a area that will have unique 
affordances for some classes of application. For instance, cubic tiles packed into the 
shape of a large cube can display, using the six faces of the larger cube, the various 
projections that could be rendered by viewing a 3D CAD drawing from each degree of 
freedom. 

As will be appreciated, display tiles need not be recti-linear but can be 
hexagonal, circular, or of arbitrary shape and size. The size of tiles need not be 
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constant for all tiles within a larger structure. Tiles need not be aligned but may require 
proximity to indicate adjoining points or edges. 

Tiles need not be in physical contact to define they are joined in a group activity 
5 but instead this may be a programmed function. The connectivity can also be 
discovered through a radio network from a coordinating server or from a distributed 
algorithm that draws in as many computers as necessary for a task, using the wireless 
network to negotiate the resources it needs. The tiled computers may also be 
connected by wired networking systems, although in cases where the topology might 

10 need to rapidly change, this is not as desirable as a wireless system. An example of a 
wired networking system of this type is a system that uses the internet to include many 
computers in a single task, one in which each computer knows its relative position even 
though they may be separated by rooms or in the extreme cases buildings, cities or 
countries. However, in general the most useful case of display tiling is when the tiles 

15 are proximate enough that they can all be viewed by one person to create a display 
medium whereby the viewing experience is an enhancement over using a single 
display. Accordingly, tiles can operate as either a single larger contiguous structure or 
they can retain individual properties and independent functions or a combination of 
both, (e.g., jumbo-tron like function, 12x12 small displays as used in a TV studio , or 

20 picture in-picture features found in commercial TVs or editing suites). 

Depending on the type of packing, various schemes can be used to allow for 
permanent, intermittent, or even one-time communication between display tiles. For 
example, close packed tiling can make use of wired connectivity between computers or 

25 could use a variety of wireless or optical communication technologies. In the case of 
wired connectivity, edge mounted conventional plug and socket connectors may be 
used to create a rigid tiled array. Plug and socket systems lend themselves to parallel 
connections for bulk and high-speed data transfer. They also provide a convenient 
method of power distribution, which can allow for one of the tiles providing a power 

30 source that supports the rest of the array. A plug and socket connection between tiled 
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computers 692 in a tiled computer array 690 (with both data and power transfer shown) 
is schematically illustrated by lines 695 in Figure 48. 

The exact design and locations of electrical connectors depends on the intended 
5 use, and subsequently , the shape of tile components. For applications where large 
seams between displays are acceptable , simple rigid connectors could be attached to 
the center of each edge, providing connectivity to all the surrounding tiles. Other 
applications might require a more complex design. For example, applications which 
require high-quality information display on a seamless array surface (i.e. tiled 
10 blueprints), might employ spring-loaded contacts on the four edge connectors. The 
spring mechanism allows all array connections to be made below the display surface, 
while tiles can still be inserted and removed from the interior of the array. Removal of 
tiles could be triggered by some host signal which actuates a release of the spring and 
pops the tile out of the array. 

15 

Serial connectivity can also be used in a close packed arrangement such as that 
illustrated in Figure 43. It has the advantage that fewer connections have to be made 
and in practice it might be more reliable. However the net bandwidth between tiles will 
be less than a parallel system. Serial communication lends itself to optical and wireless 

20 systems thus removing the need for any physical connections. For optical technologies 
alignment of the transmitter and receiver is still important although the clever use of 
light pipes and lens capture techniques can introduce more flexibility. Wireless systems 
can use many different bands of the EM spectrum (kHz, MHz, GHz), utilizing a variety 
of modulation techniques (amplitude modulated, frequency modulated, or those based 

25 on code division multiple access) and operate at a range of transmitter powers. There is 
no longer a need for direct alignment if the system is designed with suitable 
communication tolerances. The transmitter range plays a crucial part in the design. If 
there is only enough power in an emitted signal to be picked up within a few millimeters 
of a tile edge, then the signals will be isolated, the topology will be defined by the 

30 physical connectivity and the complexity of designing the system to avoid interference 
from neighboring signal sources will be minimized. However, an alternative design is to 
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use more powerful radios. In this case all tiles will be able to contact all other tiles and 
inter-tile connectivity needs to be defined by another parameter. Signal strength can be 
used or, more deliberately, information that relates the ID of a tile to a spatial map 
(perhaps held in one master tile) describing the position of all tiles in the tile array. In 

5 this system it is also necessary to minimize inter-tile interference. For digital packet- 
data systems that operate at the same frequency, carrier sense multiple access 
(CSMA-CD or CSMA-CA) systems are well known techniques to solve this problem. 
Other solutions involve tiles using different frequencies, with frequencies reused 
depending on the power of the transmitters. This is the technique used by traditional 

io cellular telephones. Yet another approach is to use Code Division Multiple Access 
(CDMA) that relies on the overlaying of signals in the same region of the EM spectrum, 
a technique known as spread spectrum modulation. 

For loose packed display tiles, the wireless techniques described above in 
is connection with close packed tiling generally become essential to the implementation. 
However, a special case of loose packing exists in which each edge of a tile does make 
contact with every other surrounding tile, except that it may only be a single point of 
contact and not at a accurately defined place. A wired version of this system can be 
built in which the entire edge of each tile is a serial connection including one of the two 
20 vertices that define the edge. Communication in each direction can be achieved by a 
variety of commercially available techniques, including use of a one-wire interface (plus 
a ground) for bidirection communication between tags and a reader. Note that the 
ground connection for a tile arrangement can be derived by sharing a common ground 
connection through the surface the tiles are laid out on. For example, the surface could 
25 be made of a metal sheet. The system may be further enhanced by ensuring that the 
edge contact is made of a magnetic material and the vertices have a magnet 
embedded in the end. Such an arrangement ensures that there will be a good electrical 
contact between the transmitter and the receiver. 

30 The receiver can also derive power from an electrically transmitted signal by 

bridge-rectifying it and storing the collected charge in a capacitor for use by its own 
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electronics. Thus, power distribution can also be included in a one-wire interface. In this 
way flexible connectivity can be achieved to support the rapid and convenient 
rearrangement of tiles in which only the minimum amount of care needs to be taken in 
setting up connectivity. 

5 

Loosely packed tile displays such as illustrated in Figure 45 may require that the 
display surfaces use a best effort algorithm to present a unified display with ail the 
sections of the displayed image bearing the correct spatial arrangement to each other 
even though, in the case of rectangular tiles, they might not be registered vertically or 

10 horizontally and have an offset angle relative to each other. In order to implement the 
desired tile display algorithm, not only is the relative arrangement of tiles important but 
the exact offset (distance and angle) from each other is also important. There are 
several methods that are suitable for automatically determining offsets between loosely 
packed tiles. For example, as illustrated with respect to Figure 46 (showing 

15 communicating tiles 652 and 654, with respective displays 651 and 653), optical 
encoding 660 along an edge can be used to identify tile orientation. It is possible to use 
a binary coded optical pattern along an edge which is regular and encodes the distance 
from a vertex at any point. A tile that abuts, or is relatively aligned, can read this pattern 
using optical sensors 658 and 659 and determine the display offset in the direction of 

20 the edge. Alternatively, as illustrated by Figure 47 (showing communicating tiles 672 
and 674), it is possible to use radio based techniques that rely on signal strength 
triangulation. Each vertex 675 or 676 of a tile 672 can contain a wireless transmitter and 
a receiver. If these vertices transmit a short characteristic wireless signal at well known 
times, a nearby tile 674 can use the receivers situated at its own vertices 680, 681 , 682, 

25 and 683 to triangulate the position of each transmitting vertex relative to them by 
measuring the relative delay by which the signal was received. After two of the vertices 
of the original transmitting tile 672 have signaled, the adjoining tile 674 can determine 
its exact position and orientation within their local region of a tile array. The transmitting 
and receiving tiles can now swap roles, with the result that both tiles will know their 

30 relative position. This process can be extended across the tile array. 
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Free format tiling differs from loose-packed tiling in that there are no proximity, or 
regular format, constraints. To demonstrate how a free format tiling system might work, 
the following example is described. Imagine a number of laptop computers each fitted 
with a Global Positioning System (GPS) and a radio modem. Each laptop can 
determine its position (with acceptable error) and communicate it to all the other laptops 
by contacting them through the radio modem. After some period of time the laptops will 
all know their relative location and absolute location. If any computers changes its 
location, it can update the local neighbors to ensure there is enough understanding in 
the array to consider the computers are in a known tiling configuration even though they 
are not close to each other and in fact they may be in different geographic regions. An 
application that might use this free format tiling system is one that wishes to ensure 
information is being correctly sent in a uniform and dispersed fashion across a very 
large area. For example, the information that each laptop computer receives could be 
instructions to release an amount of a pesticide used to control an insect that does 
damage to commercial crops. If the pesticide is released in too high a concentration in 
any locality, it may be hazardous to human health. The tiled approach allows the 
computers to roam (e.g. in the back of a truck), displaying information as to the type 
and concentration of pesticide that is to be released given their current relative 
proximity. 

Various conventional algorithms can be used to support distribution of 
information between autonomous tiled displays. These algorithms assume a system by 
which there is a master controller generating data to be displayed. There is also a large 
array of tiled computers that the system will use to display visual data and/or process 
information. Each computer in the tiled array contains a unique ID. It is the job of the 
master to split the data into pieces that each computer can display and for this 
information to be packaged along with the ID of the target computer. The algorithms 
below describe how the information travels from the master to a destination display tile: 

Daisy chain routing 
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Display tiles are arranged to have a logical connectivity with each other so that 
each one only transmits information to the next in a predefined line. The computers are 
said to be daisy chained to each. Any information sent to the start of the chain contains 
an ID and the first computer in the chain compares it to its own ID. If it matches, it acts 
5 on the data. If not, it sends the data on to the next computer in the chain until it finds its 
destination. 

N-ary routing 

In N-ary routing the path to the destination is contained in the ID of the device. A 
io simple routing example is schematically illustrated with refrence to the direction arrows 
695 in Figure 48, which shows quaternary routing in physically connected tileable 
displays. In quaternary routing, an array is conceptually arranged as a quaternary tree 
with each node having an input and three outputs. In this system each pair of bits of the 
ID contains a routing command. A 0 indicates send the packet to the first output, a 1 
15 indicates the second output, a 2 indicates the third output, and 3 indicates no further 
transmission. There is also a count that gets decremented by each node to tell 
successive nodes the bit number that is currently being considered and when the 
packet has reached its destination. In this way packets are forwarded from node to 
node with a simple choice at each stage until they reach their target display. As will be 
20 appreciated, it is possible to design N-ary systems with more than three outputs (a 
power of two is usually convenient for implementation (e.g. 4, 8 , 16 ...)). 

Flooding 

Flooding has no predefined routing structure. The computer that takes the first 
25 packet from the master checks to see if it has the correct ID. If not, the packet is sent 
out on all links to which it has not yet sent or received that packet. The result is a flood 
of copies of the packet across the array eventually reaching its destination. The packets 
must also have a maximum hop count to ensure they eventually are removed from the 
system. The disadvantage of this approach is that many more tiles are burdened with 
30 the processing of unwanted data than in the previous two schemes, which may impact 
the overall efficiency of the system. 
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Hot Potato 

The Hot Potato algorithm is similar to the Flooding algorithm, except a 
retransmitted packet is sent out on only one output that is either chosen randomly or is 
the least busy. The process stops when the packet reaches the correct tile. The time 
that a packet takes to reach its destination is not deterministic. 

As those skilled in the art will appreciate, other various modifications, extensions, 
and changes to the foregoing disclosed embodiments of the present invention are 
contemplated to be within the scope and spirit of the invention as defined in the 
following claims. 
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The claimed invention is: 

1 . A method for inputting information to a device connected to a deformable 
5 piece, the method comprising the steps of 

manipulating the deformable piece to provide a first morpheme input to the 
device, the first morpheme input normally triggering a first default action by the device, 

io asynchronously manipulating the deformable piece to provide a second 

morpheme input to the device, with the second morpheme input converting the normally 
triggered first default action to a second action. 

2. The method of claim 1, further comprising the step of spatially 
15 manipulating the deformable piece as at least one of the first and second morpheme 

inputs to the device. 

3. The method of claim 1, wherein at least one of the first and second 
morpheme inputs to the device requires spatially translating the device. 

20 

4. The method of claim 1, wherein at least one of the first and second 
morpheme inputs to the device requires panning the device. 

5. The method of claim 1, wherein at least one of the first and second 
25 morpheme inputs to the device requires pushing the device relative to a user. 

6. The method of claim 1, wherein at least one of the first and second 
morpheme inputs to the device requires pulling the device relative to a user. 

30 7. The method of claim 1, wherein at least one of the first and second 

morpheme inputs to the device requires moving the device within a room. 
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8. The method of claim 1, wherein at least one of the first and. second 
morpheme inputs to the device requires moving the device between widely separated 
sites 

9. The method of claim 1, wherein at least one of the first and second 
morpheme inputs to the device is based on detected light variations. 

10. The method of claim 1, wherein at least one of the first and second 
morpheme inputs to the device is based on detected thermal variations. 

11. The method of claim 1, wherein at least one of the first and second 
morpheme inputs to the device is based on detected electromagnetic variations. 

12. The method of claim 1, wherein at least one of the first and second 
morpheme inputs to the device is based on detected vibration variations. 

13. The method of claim 1, wherein at least one of the first and second 
morpheme inputs to the device is based on detected acoustic variations. 

14. A method for inputting information to a device having a processor and an 
integrally connected deformabie piece, the method comprising the steps of 

shaking the deformabie piece to provide a morpheme input to the device, and 
triggering a first default action by the device in response to shaking the 
deformabie piece. 

15. A method for inputting information to a device having a processor and an 
integrally connected deformabie piece, the method comprising the steps of 

flicking the deformabie piece to provide a morpheme input to the device, and 
triggering a first default action by the device in response to flicking the 
deformabie piece. 
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16. A method for inputting information to a device having a processor and an 
integrally connected deformable piece, the method comprising the steps of 

facing the deformable piece to provide a morpheme input to the device, and 
triggering a first default action by the device in response to facing the deformable 

piece. 

17. A method for inputting information to a device having a processor and an 
integrally connected deformable piece, the method comprising the steps of 

lifting the deformable piece to provide a morpheme input to the device, and 
triggering a first default action by the device in response to lifting the deformable 

piece. 

18. A method for inputting information to a device having a processor and an 
integrally connected deformable piece, the method comprising the steps of 

orienting the deformable piece relative to an external environmental object to 
provide a morpheme input to the device, and 

triggering a first default action by the device in response to orienting the 
deformable piece relative to an external environmental object. 

19. A method for inputting information to a device having a processor and an 
integrally connected deformable piece, the method comprising the steps of 

whacking the deformable piece to provide a morpheme input to the device, and 
triggering a first default action by the device in response to whacking the 
deformable piece. 
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Abstract of the Disclosure 

A method for inputting information to a device connected to a deformable piece 
includes the steps of manipulating the deformable piece to provide a first morpheme 
input to the device. The first morpheme input normally triggers a first default action by 
the device, such as controlling a display, modifying a data structure, or communicating 
with another electronic device. When a user asynchronously manipulates the 
deformable piece to provide a second morpheme input to the device, the second 
morpheme input converts the normally triggered first default, action to a second action 
generally not equivalent to the first default action. This mode of interaction allows 
formation of morphemic sentences to control a graspable device. 
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PATENT APPLICATION 

Attorney Docket No. D/97403 

DECLARATION AND POWER OF ATTORNEY FOR PATENT APPLICATION 

As a below named inventor, i hereby declare that: 

My residence, post office address and citizenship are as stated below next to my name, 

I believe i am the original, first and sole inventor (if only one name is listed below) or an original, first and 
joint inventor (if plural names are listed below) of the subject matter which is claimed and for which a patent 
is sought on the invention entitled: A COMPUTER USER INTERFACE USING A PHYSICAL 
MANIPULATORY GRAMMAR 

the specification and claims of which 

□ are attached hereto OR S was filed on 8/29/1997 as U.S. Application No. 08/920.443 

I hereby state that I have reviewed and understand the contents of the above-identified specification, 
including the claims. 

I acknowledge the duty to disclose information which is material to the patentability as defined in Title 37, 
Code of Federal Regulations, §1.56. 

I hereby claim priority benefits under Title 35, United States Code, §119 of any foreign or U.S. Provisional 
application(s) for patent listed below, and have also identified below any foreign application(s) or Provisional 
application(s) for patent having a filing date before that of the application on which priority is claimed: 

Prior Foreign or U.S. Provisional Application(s) 



(Number) 



(Country) 



(Day/Month/Year Filed) 



POWER OF ATTORNEY: As a named inventor, I hereby appoint the following registered 
practitioners to prosecute this application and transact all business in the Patent and Trademark 
Office connected therewith. 

Ronald Zibelli, Reg. No. 20,484; Henry Fleischer, Reg. No. 25,582; 

John E. Beck, Reg. No. 22,833; Mark Costello, Reg. No. 31,342; 

Serge Abend, Reg. No. 24,152; Robert A. Burtzlaff, Reg. No. 35,466 
Richard B. Domingo, Reg. No. 36,784; 



ADDRESS ALL CORRESPONDENCE TO: 
Ronald Zibelli 
Xerox Corporation 
Xerox Square 20A 
Rochester, New York 14644 



DIRECT TELEPHONE CALLS TO: 
(name and telephone number) 
Robert A. Burtzlaff 
650-812-4268 



I hereby declare that all statements made herein of my own knowledge are true and that all statements made 
on information and belief are believed to be true; and further that these statements were made with the 
knowledge that willful false statements and the like so made are punishable by fine or imprisonment, or both, 
under §1001 of Title 18 of the United States Code and that such willful false statements may jeopardize the 
validity of the application or any patent issuing thereon. 
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DECLARATION AND POWER OF ATTORNEY, continued 

Name of sole or first inventor: Kenneth P. Fishkin 



Inventor's Signature: 

Residence: 924 Haven Avenue, Redwood City, California 94063 
Citizenship: USA 
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Date: 



J fa' A 




Post Office Address: 
(Same as above) 



Name of second joint inventor: Bevedy L Harrison 




Inventor's Signature: _ _ . 

Residence: 720 College Avenue/Palo Alto,^a1ifornia 94306 
Citizenship: Canada 



Date: J? /^A & 

Post Office Address: 
(Same as above) 



Name of third joint inventor: 



Inventor's Signature: 
Residence: 1 541 Morton Avenue, Los Altos, California 94024 
Citizenship: United Kingdom 




Date: 



Post Office Address: 
(Same as above) 



-2- 



